A unified view of TD algorithms, introducing Full-gradient TD and Equi-gradient descent TD

This paper addresses the issue of policy evaluation in Markov Decision Processes, using linear function approximation. It provides a unified view of algorithms such as TD(λ), LSTD(λ), iLSTD, residual-gradient TD. It is asserted that they all consist in minimizing a gradient function and differ by the form of this function and their means of minimizing it. Two new schemes are introduced in that framework: Full-gradient TD which uses a generalization of the principle introduced in iLSTD, and EGD TD, which reduces the gradient by successive equi-gradient descents. These three algorithms form a new intermediate family with the interesting property of making much better use of the samples than TD while keeping a gradient descent scheme, which is useful for complexity issues and optimistic policy iteration. 1 The policy evaluation problem A Markov Decision Process (MDP) describes a dynamical system and an agent. The system is described by its state s ∈ S. When considering discrete time, the agent can apply at each time step an action u ∈ U which drives the system to a state s = u(s) at the next time step. u is generally non-deterministic. To each transition is associated a reward r ∈ R ⊂ R. A policy π is a function that associates to any state of the system an action taken by the agent. Given a discount factor γ, the value function v of a policy π associates to any state the expected discounted sum of rewards received when applying π from that state for an infinite time: